Aug 11
17
Posted by Lindsay As you’ve probably learned, you can’t always rely on search engine spiders to do an effective job when they visit and index your website. Left to their own devices bots can generate duplicate content, perceive important pages as junk, index content that shouldn’t serve as a user entry point, and many other issues. There are a number of tools at our disposal that allow us to make the most of bot activity on a website such as the meta robots tag, robots.txt, x-robots-tag, canonical tag and others.
Excerpt from:
Robot Access & Indexation Restriction Techniques: Avoiding Conflicts